1.1 Learning by Reward

Imagine you are trying to train a group of monkeys to play basketball. Every time a monkey makes a successful shot, you give it a banana. If the monkey misses the shot, you do not give it a banana. Over time, the monkeys start to associate making successful shots with receiving a banana and start to develop strategies to improve their chances of making a successful shot, such as passing the ball to a teammate or practicing their aim.

Better still, Imagine you are training a robot to do laundry. You set up a reinforcement learning algorithm where the robot receives a positive reward (high five) when it successfully sorts the clothes, adds the correct amount of detergent, and washes the clothes without damaging them. The robot receives a negative reward (no high five) if it damages the clothes, spills detergent, or makes a mistake in the sorting process. At first, the robot has no idea how to do laundry.

Source: Robot Laundry GIF

It may accidentally mix whites and colors or add too much detergent, resulting in negative rewards. However, through trial and error, the robot gradually learns the correct sequence of actions to take when doing laundry. As the robot continues to do laundry over time, the robot starts to associate successfully washing and drying laundry with receiving a high-five and starts to optimize its laundry skills to maximize the chances of getting a high-five. It refines its strategy and learns to adapt to several types of fabrics and stains. It may adjust the water temperature, or the amount of detergent based on the type of fabric being washed or selecting the correct cycle.

Over time, the robot becomes your skilled laundry assistant, capable of doing laundry efficiently and effectively. Now imagine that you change the scenario by adding a new type of fabric or stain that the robot has never encountered before. The robot may initially struggle to determine the correct cycle or water temperature or detergent amount to use. However, by experimenting with different settings and observing the results, the robot can learn to adjust its strategy to handle the new fabric or stain type. These are just a few examples of reinforcement learning. The robot/monkeys are the agents, the shot/laundry task are the actions, the banana/high-five are the rewards, and the association between the action and the reward is the policy. By using a reward signal to guide the agents’ behavior, you can train them to learn a specific task or behavior. This also demonstrates how reinforcement learning algorithms can adapt to new and unexpected situations, even in complex tasks like doing laundry.

1.2 Stock Trading

How can we teach a 10-year-old to trade stocks? First of all, we start with basics of risks and rewards. Tell them to forget about lemonade stands and allowance money, we are talking real cash here. Let them imagine playing a game where they get to buy and sell imaginary items, like Pokemon cards or Beanie Babies. They should want to buy low and sell high to make a profit, just like in real life. The game is called RL, that's right, it stands for "Real" Life and the stakes are high, but the rewards can be even higher.

Source: JPMorgan

Now, here’s where things get tricky. RL isn’t like other games where they can just hit “reset” and start over if you mess up. They have to learn from their mistakes and make smarter choices next time. So, one should do their research, pay attention to trends, and don’t be afraid to take risks (but not too many!). It’s all about balancing risk and reward, just like in RL.

In the world of stock trading, timing is everything. It is no secret that the stock market is unpredictable, with prices rising and falling based on everything from global events to social media trends. Recently, stocks like Bitcoin, Dogecoin, GameStop and Tesla have captured the attention of investors and the public alike^[1]. Whether it is buying low and selling high or knowing when to hold onto a stock for the long term, making the right decision can mean the difference between profit and loss. But how can traders stay on top of market trends and make informed decisions in such a volatile environment? Is it possible to automate trading entirely for traders who want to trade 24/7 but do not want to be glued to their screens all the time? One answer is reinforcement learning, a form of machine learning that trains algorithms to learn from past actions and outcomes and make better decisions over time^[2,3]. In this project, we will explore the application of reinforcement learning in stock trading and how this can potentially change the way we think about financial investment decisions.

2.1 Mathematical Nature of Reinforcement Learning

We can illustrate the nature of reinforcement learning as follows: at each sequence of discrete time points (or steps), with t = 0, 1, 2, 3, etc., the agent and environment interact. The agent receives information about the environment’s state, \(s_t\) within S, at each time point/step, t, where S represents the set of possible states, which on this basis, selects an action from A(\(s_t\)), the set of actions to choose from. The agent receives the numerical reward within R, and then finds itself under a new state^[8]. The diagram below illustrates this process:

The goal of reinforcement learning (RL) can be summarized as finding a policy that maximizes the expected cumulative reward over a sequence of interactions between an agent and its environment. Mathematically, this can be expressed as:

\[\boldsymbol{maximize~ E[R] = \max_{\pi} E\left[\sum_{t=0}^{\infty} \gamma^t r_t\right]}\]

where \(\boldsymbol{E[R]}\) is the expected cumulative reward, \(\boldsymbol{\gamma}\) is the discount factor (\(0 \leq \gamma \leq 1\)), \(\boldsymbol{r_t}\) is the reward obtained at time \(\boldsymbol{t}\), and the summation is taken over all time steps from \(\boldsymbol{t=0}\) to \(\boldsymbol{t=\infty}\).

The objective function in RL is typically the expected cumulative reward, which is also known as the return. This function is used to evaluate the performance of a policy, which maps states to actions. The objective is to find the policy that maximizes the expected return over a given time horizon.

The objective function can be expressed as:

\[\boldsymbol{J(\pi) = E_{\pi}\left[\sum_{t=0}^{\infty} \gamma^t r_t\right]}\]

where \(\boldsymbol{J(\pi)}\) is the expected return for policy \(\boldsymbol{\pi}\), \(\boldsymbol{E_{\pi}}\) is the expectation over the distribution of trajectories generated by following policy \(\boldsymbol{\pi}\), and the summation is taken over all time steps from \(\boldsymbol{t=0}\) to \(\boldsymbol{t=\infty}\).

The goal of RL is to find the policy \(\boldsymbol{\pi^*}\) that maximizes the objective function \(\boldsymbol{J(\pi)}\), i.e.,

\[\boldsymbol{\pi^* = \arg\max_{\pi} J(\pi)}\] Since the challenge that remains is to nudge the policy parameters such that the return can be maximized, we can meet this goal by using the gradient of the objective function. This is known as vanilla gradient ascent. We should note that gradient descent and gradient ascent are synonymous, with only a sign change being the notable difference (sign change is moved to downhill for gradient descent)^[9].

Policy Gradient Theorem

• In step 1, we observe that our goal is to find the gradient of the objective function.
• Then, within step 2, the expected return is replaced with an implementation (can be any of your choice), however it is usually easier to proceed with the action-value function. We denote the expected return as equivalent to each action’s (Q) value multiplied by the probability that the particular action is visited (which is defined by the policy, \(\pi\)).
• We take the gradient with respect to the parameters of the policy. Therefore, we can move Q, the action value function, outside of the calculation of the gradient (as defined in step 3).
• Proceeding with our steps, for step 4, we multiply the entire equation by \(\pi\)(a | s)/\(\pi\)(a | s), which is equivalent to one. Therefore, the equation is not altered.
• For step 5, we proceed with a typical mathematical identity, which replaces the fraction with a natural logarithm. This is a fairly standard computational trick, where we prevent the multiplication of small probabilities, which can make the gradients zero (also known as the vanishing gradient problem).
• Finally, for step 6, we can remember that the summation of action values, which are multiplied by the probabilities for the action, are what we started with in step 2. However, as a substitute to summing over the actions, we instruct the agent to adhere by the policy and return the expectation of the specified action over time^[9].

2.2 Key Elements of Reinforcement Learning

• Agent: The agent in RL is the decision-making entity that interacts with the environment. In the stock trading analogy, the agent could be the trader who decides when or which stocks to buy and sell.

• Environment: The environment in RL is the world in which the agent operates and interacts with. In the stock trading analogy, the environment would be the stock market, which includes all the stocks, their prices, and the various events that affect the market.

• State (S): A state in RL refers to the current situation of the environment, as observed by the agent. In the stock trading analogy, a state would include the current prices of the stocks the trader is interested in, the current state of the economy, and any other relevant factors that may influence the decision to buy or sell.

• Policy (\(\pi\)): A policy in RL is a set of rules or instructions that the agent uses to determine its actions in a given state. In the stock trading analogy, a policy could be the algorithm that the agent uses to determine which stocks to buy and sell based on the current state.

• Action (A): In RL, the action is the decision made by the agent based on the current state and the policy. In the stock trading analogy, the action would be the trader’s decision to hold, buy or sell a specific stock.

• Reward (R): A reward in RL is a feedback signal that the agent receives after taking an action. It indicates how good or bad the action was in achieving the agent’s objective. In the stock trading analogy, the reward would be the profit or loss made from the trade.

• Penalty: A penalty in RL is a negative feedback signal that the agent receives after taking an action that is not desired or expected. In the stock trading analogy, the penalty would be the loss incurred from a bad trade or missed opportunity..

2.3 Advantages of Reinforcement Learning

• Learning through trial and error: Unlike supervised learning, where the algorithm is provided with labeled data, RL algorithms learn by trial and error, without explicit supervision thus making it well-suited for applications where it is difficult or impractical to provide labeled data.

• Flexibility: RL can be used in both single-agent and multi-agent settings. In a single-agent setting, the agent learns to optimize its behavior by interacting with the environment while in a multi-agent setting, multiple agents learn to interact with each other and optimize their behavior collectively.

• Adaptability: RL algorithms can adapt to changes in the environment and adjust their behavior accordingly. This makes RL well-suited for dynamic and uncertain environments where traditional machine learning approaches may struggle.

• Exploration: RL algorithms are designed to explore new strategies and actions in order to maximize their rewards. This can lead to the discovery of novel solutions and approaches that may not have been considered otherwise.

2.4 Challenges of Reinforcement Learning

• Exploration-exploitation tradeoff: RL algorithms need to balance between exploring the environment to find new and potentially better actions, and exploiting the knowledge they already have to maximize rewards in the past. This tradeoff can be challenging, especially in complex environments with many possible actions.

• Generalization: RL algorithms may struggle to generalize their learned policies to new, unseen environments or tasks that were not encountered during training, which can limit their usefulness in practice. Data efficiency: RL algorithms typically require a large amount of data to learn a good policy, which can be time-consuming and expensive to obtain.

• Reward engineering: The quality of the learned policy depends heavily on the reward function used, which can be difficult to design in a way that accurately reflects the desired behavior.

• Safety and ethics: RL agents can learn to take actions that are harmful or unethical, especially if the reward function is not carefully designed or the agent’s behavior is not appropriately constrained.

• Interpretability: RL algorithms can be difficult to interpret and explain, especially when they use complex models or operate in high-dimensional state and action spaces.

3.1 1. Q-Learning

Q-learning^[10] is a popular value-based reinforcement learning algorithm based on the well known Bellman equation:

\[\boldsymbol{V(s) = \mathbb{E}[R_{t+1} + \gamma V(s_{t+1}) | S_t = s]}\] Where; \(\boldsymbol{V(s)}\) is the value of the current state \(s\), \(\mathbb{E}\) refers to the expectation, while \(\gamma\) refers to the discount factor that determines the importance of future rewards. From the above definition of the Bellman’s equation, the action value function can be expressed as:

In Q-learning, the agent uses an iterative approach to update the Q-function estimates and learn an estimate of the optimal action-value based on the rewards obtained from each function. The optimal Q-value, denoted as Q* can be expressed using the law of total probability as:

\[\boldsymbol{Q^*(s, a) = \mathbb{E}{s' \sim p(\cdot|s,a)} \Big[r + \gamma \max_{a'} Q^*(s', a')\Big | s,a\Big]}\] where \(\boldsymbol{s'} \sim \boldsymbol{p(\cdot|s,a)}\) denotes the next state sampled from the transition probability distribution, \(\boldsymbol{p(\cdot|s,a)}\), \(\boldsymbol{r}\) is the immediate reward obtained after taking action \(\boldsymbol{a}\) in state \(\boldsymbol{s}\).

In the context of stock trading, Q-learning can be used to learn the optimal buying and selling decisions for a given stock, based on historical price data and other market indicators. The agent can learn to maximize its expected profit over a given time horizon, taking into account the risks and uncertainties associated with stock trading.

The Q-learning algorithm can be described by the following update rule:

\[\boldsymbol{Q(S_t, A_t) \leftarrow Q(S_t, A_t) + \alpha \Big[R_{t+1} + \gamma \max_a Q(S_{t+1}, a) - Q(S_t, A_t)\Big]}\]

where:

• \(\boldsymbol{Q(S_t, A_t)}\) is the current estimate of the Q-value for state-action pair - \((\boldsymbol{S_t}, \boldsymbol{A_t})\),
• \(\boldsymbol{R_{t+1}}\) is the immediate reward obtained after taking action
• \(\boldsymbol{A_t}\) in the next state \(\boldsymbol{S_{t+1}}\),
• \(\boldsymbol{\alpha}\) is the learning rate,
• \(\boldsymbol{\gamma}\) is the discount factor, and
• \(\boldsymbol{\max_a Q(S_{t+1}, a)}\) is the maximum Q-value over all possible actions in the next state \(\boldsymbol{S_{t+1}}\).

The algorithm is summarized as below:

3.2 2. State-Action-Reward-State-Action (SARSA)

SARSA (State-Action-Reward-State-Action) is a RL algorithm that is used for online and on-policy learning. It is similar to Q-learning, but instead of updating the Q-value of the current state-action pair using the maximum Q-value of the next state, SARSA updates the Q-value using the Q-value of the next state-action pair, that is, it learns the Q-value of the policy being followed^[2]. The SARSA algorithm can be described by the following update rule:

\[\boldsymbol{Q(S_t,A_t) \leftarrow Q(S_t,a_t) + \alpha[r_{t+1} + \gamma Q(S_{t+1},a_{t+1}) - Q(S_t,a_t)]}\]

Where:

• \(\boldsymbol{Q(S_t,a_t)}\) is the Q-value of the current state-action pair \((S_t,A_t)\)
• \(\boldsymbol{\alpha}\) is the learning rate that determines the impact of the new information on the existing Q-value
• \(\boldsymbol{r_{t+1}}\) is the immediate reward obtained after taking action \(A_t\) in state \(S_t\)
• \(\boldsymbol{\gamma}\) is the discount factor that determines the importance of future rewards
• \(\boldsymbol{Q(S_{t+1} ,a_{t+1}})\) is the Q-value of the next state-action pair \((S_{t+1},a_{t+1})\) under the current policy. This is the value of the action selected by the agent in the next state.
• \(\boldsymbol{Q(S_t,a_t)}\) is the current estimate of the Q-value of state-action pair \((S_t,a_t)\).

This equation updates the estimate of the Q-value of the current state-action pair by adding the difference between the observed reward and the estimate of the next state-action pair’s Q-value, multiplied by the learning rate \(\alpha\).

The algorithm is summarized as below:

3.3 3. Deep Q-network (DQN)

Deep Q-Network (DQN), is a type of RL algorithm that was introduced in 2015 and uses deep neural networks to approximate the Q-values of state-action pairs in a Markov decision process^[11].

The DQN algorithm uses experience replay and target networks to improve the stability and efficiency of the learning process. Experience replay involves storing the agent’s experiences (i.e., state, action, reward, next state) in a replay buffer and sampling mini-batches of experiences randomly from the buffer to train the neural network.

Target networks involve creating a separate neural network with the same architecture as the Q-network but with frozen weights to estimate the target Q-values used in the Bellman equation. This reduces the correlation between the target and predicted Q-values, which improves the stability of the learning process.

The update equation for Deep Q-Network (DQN) algorithm can be written as follows:

\[\boldsymbol{Q(s,a) = Q(s,a) + \alpha [r + \gamma \max_{a'} Q'(s',a') - Q(s,a)]}\]

Where:

• \(\boldsymbol{Q(s,a)}\) is the estimated Q-value for state s and action a;
• \(\boldsymbol{r}\) is the reward obtained after taking action \(a\) in state \(s\);
• \(\boldsymbol{s'}\) is the next state after taking action \(a\) in state $s;
• \(a'\) is the action with the highest Q-value in state \(s'\);
• \(\boldsymbol{Q'(s',a')}\) is the target Q-value for the next state-action pair, computed using a separate target network that is periodically updated with the main network;
• \(\boldsymbol{\alpha}\) is the learning rate;
• \(\boldsymbol{\gamma}\) is the discount factor, which determines the weight given to future rewards.

The algorithm is summarized as below:

5.1 Implementation of Q-learning

Our code implements the Q-learning algorithm to create a simple trading strategy for stocks. The Q-learning algorithm is a model-free reinforcement learning technique used to learn an optimal policy based on trial and error. It involves building a Q-table that maps the current state and action to the expected future reward. In this case, the state is the current stock price, and the action is either to buy or hold/sell.

The function ‘Q_learning’ takes a data frame of stock prices and applies the Q-learning algorithm to each stock to create a Q-matrix that stores the expected future rewards for each state-action pair. It also creates a list of Boolean Q-tables for each stock, which specify whether to buy or hold/sell at each time step based on the Q-matrix. Finally, it returns the Q-matrix, actions taken, and Q-table for each stock.

Our code then loads stock data for four companies “AAPL”, “AMZN”, “JNJ”, “NFLX”, extracts the adjusted prices, and combines them into a matrix. It performs exploratory data analysis (EDA) by plotting the distribution of prices for each stock. It then applies the ‘Q_learning’ function to each stock to create a list of Q-matrices, actions taken, and Q-tables.

The ‘Q_learning_summary’ function generates a summary table of the actions taken for each stock, converts the action codes to action names, and splits the summary table by stock. It then creates tables for action for each stock and access the Q matrix for each stock. The ‘predict_prices’ function predicts the prices of a stock using the Q matrix and the current prices of the stock.

5.2 Implementation of SARSA

The SARSA code provided is an implementation of the SARSA (State-Action-Reward-State-Action) reinforcement learning algorithm to train an agent to make buy, sell, or hold decisions for the stocks used in this project. The code is set up to train the agent on a subset of the data (2010-2016) and test the agent on the remaining data (2017-2022).

# extract closing prices from data
closing_prices <- train$close

# define state based on previous 3 closing prices
define_state <- function(closing_prices, i) {
  if (i < 4) {
    # if not enough data points yet, use the first one as the state
    state <- closing_prices[1]
  } else {
    # use previous 3 closing prices as the state
    state <- closing_prices[(i-3):i]
  }
  return(state)
}

Next, we defined the tune function that takes in a dataset, a range of values for the learning rate (alpha), discount factor (gamma), and epsilon-greedy policy (epsilon). The function uses nested loops to iterate over each combination of alpha, gamma, and epsilon values and runs the SARSA algorithm on the dataset using those hyperparameters. The function then calculates the total reward obtained by the SARSA algorithm and stores the hyperparameters and the total reward as a list in the “results” variable.

After performing the hyperparameter tuning, the optimal combination of hyperparameters that yielded the highest total reward for the SARSA algorithm were; \(\alpha= 0.8\), \(\gamma= 0.3\), and \(\epsilon =0.1\). These were associated with the highest total reward of 21069.54.

# function that extracts the hyperparameters associated with the best result and returns them as a list

tune <- function(data, alpha_range, gamma_range, epsilon_range) {
  # initialize results list
  results <- list()
  
  # loop through alpha, gamma, and epsilon values
  for (alpha in alpha_range) {
    for (gamma in gamma_range) {
      for (epsilon in epsilon_range) {
        
        # run SARSA algorithm with current hyperparameters
        trades_and_Q <- sarsa(data, alpha = alpha, gamma = gamma, epsilon = epsilon)
        
        # calculate total reward and store results
        total_reward <- sum(unlist(lapply(trades_and_Q$trades, function(x) x$reward)))
        results[[length(results)+1]] <- list(alpha = alpha, gamma = gamma, epsilon = epsilon, total_reward = total_reward)
      }
    }
  }
  
  # return hyperparameters with highest total reward
  best_result_idx <- which.max(sapply(results, function(x) x$total_reward))
  best_result <- results[[best_result_idx]]
  
  return(list(alpha = best_result$alpha, gamma = best_result$gamma, epsilon = best_result$epsilon, total_reward = best_result$total_reward))
}

We then used the combination of the best hyperparameters to train the algorithm on the train test. The figure below shows the algorithm made a higher number of buy trading actions compared to the sell actions.

To test the SARSA algorithm on the test data, we modified the sarsa function to use the learned Q-table to choose actions based on the current state instead of using the epsilon-greedy policy. Here, we removed the Q-table initialization and epsilon-greedy policy from the original sarsa function, and added an additional input parameter Q to the sarsa_test function to take in the learned Q-table. To test the SARSA algorithm on the test data, we can call the sarsa_test function with the test data and the learned Q-table as input:

The portfolio value over time represents the value of the portfolio as it changes over the period of the test data. It shows how the agent’s decisions based on the SARSA algorithm impacted the portfolio’s value. A higher portfolio value means that the agent’s decisions led to more profitable trades and a better overall outcome.

From the above results; while the code may be able to generate some predictions on the test data, it is important to keep in mind that SARSA is a model-free algorithm, meaning that it does not explicitly model the underlying dynamics of the stock market. Instead, it learns through trial and error by exploring different actions and observing the resulting rewards.

Limitation: It is difficult to make definitive predictions on the test data using this code alone. It would be more appropriate to evaluate the performance of the agent on the test data by analyzing its trades and portfolio value over time and comparing it to a benchmark such as a buy-and-hold strategy. Additionally, it would be useful to further analyze the agent’s behavior and performance by adjusting the hyperparameters (alpha, gamma, and epsilon) and potentially using alternative reinforcement learning algorithms or additional features.

5.3 Implementation of DQN

Next, we used tensorflow package in python to implement the DQN on our stock data. However, since training on the full train data set was not feasible due to computational time and memory capacity, we trained the DQN on one stock, NFLX. To reproduce the same using other stocks, all one has to do is just select the stock they want from the drop down widget created in the DQN file. We used the same train/test splits as in the previous sections. We build some of the code used in this section from Analytics Vidhya, but tailored it to serve the needs of our project.

The code in the DQN file defines a class Agent which implements a Q-learning algorithm for trading in financial markets.The __init__ method initializes the hyperparameters and the neural network architecture. act method selects an action based on the current state and get_state method returns the current state. replay method trains the Q-network. buy method buys or sells shares according to the action selected by the Q-network.

The main logic of the code is in the buy method, which takes the initial money and tries to buy and sell shares at different time steps based on the Q-network’s decisions. The actions can be to buy, sell, or do nothing. The shares are bought if the Q-network selects action 1 and if the price is less than the available initial money. The shares are sold if the Q-network selects action 2 and if there are shares in the inventory. The profit or loss is calculated based on the price at which the shares are bought and sold. The overall rewards are stored in the total profit variable.

The code also includes a neural network architecture with several hidden layers and uses the TensorFlow library for deep learning. We tested the performance of 4 different architectures, with Adam chosen as our chosen gradient descent algorithm as shown in the snip below, with a default batch size of 32. Results of our evaluation are presented below for our four network architectures:

Below are the performance results for the different architectures;

We found the best architecture to be the fourth one; the one with two neural networks, with 512 neurons in the first layer and 256 in the second layer. The metrics used to compare our models were the total gains in terms of total investment which were observed over the trading days across each 50 Episode period during training. The fourth architecture had 60.97% gains in investment during training.

Next, we tested the best network architecture on the test data. The buy function will return the buy, sell, profit, and investment figures.

# test the agent
states_buy_test, states_sell_test, total_gains_test, invest_test = test_agent.buy(initial_money = initial_money)

Below is a snip of how the agent was trading in the test data for the first 35 days.

For the Netflix stocks, the figure below shows the trading decisions made by the agent in the test data.

We observe that in the test data, the agent achieved a 14.96% gains in total investments.